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Abstract 

Background: Data on HCV-related cirrhosis progression are scarce in developing countries in general, and in Egypt 
in particular. The objective of this study was to estimate the probability of death and transition between different 
health stages of HCV (compensated cirrhosis, decompensated cirrhosis and hepatocellular carcinoma) for an 
Egyptian population of patients with HCV-related cirrhosis. 

Methods: We used the "elicitation of expert opinions" method to obtain collective knowledge from a panel of 23 
Egyptian experts (among whom 17 were hepatologists or gastroenterologists and 2 were infectiologists). The 
questionnaire was based on virtual medical cases and asked the experts to assess probability of death or probability 
of various cirrhosis complications. The design was a Delphi study: we attempted to obtain a consensus between 
experts via a series of questionnaires interspersed with group response feedback. 

Results: We found substantial disparity between experts' answers, and no consensus was reached at the end of the 
process. Moreover, we obtained high death probability and high risk of hepatocellular carcinoma. The annual 
transition probability to death was estimated at between 1 0.1 % and 61 .5% and the annual probability of occurrence 
of hepatocellular carcinoma was estimated at between 16.8% and 58.9% (depending on age, gender, time spent in 
cirrhosis and cirrhosis severity). 

Conclusions: Our results show that eliciting expert opinions is not suited for determining the natural history of 
diseases due to practitioners' difficulties in evaluating quantities. Cognitive bias occurring during this type of study 
might explain our results. 

Keywords: Delphi method, Expert knowledge elicitation, Methodological bias, Risk perception, Cognitive bias, 
HCV in Egypt 



Background 

The Delphi method is used to reach a consensus between 
experts on a specific topic via a series of questionnaires in- 
terspersed with feedback of group answers [1]. Originally 
developed by the Rand Corporation for applications in 
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forecasting the impact of technology on warfare [2], this 
method has since been applied to many fields, including 
medical research [3-6], in order to compensate for the lack 
of empirical data concerning specific topics in medical 
literature. 

The aim of this method is not to collect knowledge 
about a subject, but rather to gather opinions [7]. This 
implies that erroneous estimates might occur in evalu- 
ation of some quantities by a panel of experts even when 
a convergence of opinion is observed. In particular, 
quantities such as probability of disease progression ac- 
cording to health status involve reasoning mechanisms 
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that may produce cognitive bias. Here we demonstrate 
this, and we explain why the Delphi method may not be 
accurate when estimating probabilities of disease evolu- 
tion. We use the example of a Delphi study we conducted 
in Egypt to estimate evolution of HCV-infected patients 
with cirrhosis. 

The primary objective of this study was to use expert 
opinion to determine transition probabilities in a deci- 
sion model of natural history of HCV-related cirrhosis in 
Egypt, namely, probability of death and probability of tran- 
sition between the different stages of HCV (compensated 
cirrhosis, decompensated cirrhosis, hepatocellular carcin- 
oma (HCC), etc.). Studies of the natural history of HCV- 
related cirrhosis have previously been published [8,9], but 
they focused on populations in northern countries with dif- 
ferent genotypes (in Egypt, HCV infections are mainly 
genotype 4 [10], while in northern countries, this genotype 
is uncommon [11]); moreover, the Egyptian health care 
system differs, as does the population. In Egypt, the ab- 
sence of alcohol is favorable to patients, while co-infection 
with bilharzias, hepatitis B or overweight, increase the risk 
of complications. For these reasons, we felt that literature 
estimations were inappropriate for an Egyptian population. 

Methods 

Delphi 

The study was conducted according to the following plan: 

1. Elaboration of the questionnaire: a questionnaire 
was designed to collect the estimated probability of 
evolution from cirrhosis to complications or death. 
This questionnaire was tested on two clinicians prior 
to the first-round meeting to ensure that questions 
were sufficiently clear for the reader. 

2. Choice of experts: a panel of experts was selected 
according to two criteria. First, an equal proportion of 
professorial and non-professorial physicians, in 
order to have different profiles of experts and to 
gather information from different types and places of 
practice. Second, experts from different parts of Egypt, 
so as to avoid only specifically local information. We 
also chose experts with a wide range of ages and 
medical experience. 

3. A private one-day face-to-face meeting was 
organized with experts so as to complete the 
questionnaire (first round): after receiving instructions 
via an explanation form (see Additional file 1) and an 
oral presentation of the study, each expert completed 
the questionnaire (see Additional file 2). No indications 
of possible likely values or clues were given and no 
preliminary discussion between experts took place at 
this stage. Experts also filled out a short form describing 
their medical specialty and experience and providing 
information about their familiarity with HCV infections. 



A numeric identifier was attributed to each participant 
to ensure anonymity of results. 

4. A statistical summary of group answers was 
presented to the experts in the form of means with 
95% confidence intervals and ranges for each 
quantity of interest. Experts were able to discuss the 
different questions and to eventually disagree 
concerning initial trends in responses. This was 
done at the end of the first-round meeting 
mentioned above. 

5. Each participating expert's questionnaire was 
returned to that expert with a summary of group 
responses. Experts were then asked to change their 
responses or to maintain their first choices based on 
what had been discussed (second round). Group 
answers in this final step constituted Delphi's final 
results. 

The study did not require formal ethical approval ac- 
cording to Egyptian regulations. The participants of this 
study provided informed consent for publication. 

Questionnaire 

The questionnaire covered the natural history of HCV- 
related cirrhosis in Egyptian patients. Experts were asked 
to estimate the probability of death and/or HCC occur- 
rence within one year starting from the following health 
stages: (i) compensated cirrhosis stage; (ii) first year of 
first cirrhosis decompensation episode (i.e. ascites, di- 
gestive hemorrhage, encephalopathy, icterus); (iii) stable 
decompensated cirrhosis stage (in the years following the 
first episode of decompensation without any other decom- 
pensation); and (iv) progressive decompensated cirrhosis 
stage (in the years following successive decompensation 
episodes). Moreover, they were asked to estimate the 
probability of death in patients with HCC. These transi- 
tion probabilities from one health state to the next were 
estimated on the base of virtual cases according to the fol- 
lowing patient characteristics: gender, age and time spent 
in current disease stage. The rank of each question is 
given in Table 1. 

Experts were able to respond to each question by spe- 
cifying: (i) a single point-estimate; or (ii) a range with a 
lower and higher estimate. The questionnaire is available 
in the Additional file 2. 

Statistical analysis 

A descriptive analysis was performed to summarize an- 
swers from each round. Means, 95% confidence intervals 
(95% CI) and interquartile ranges (IQR) for each value 
were estimated. When the expert's answer was a range 
rather than a point estimate, we chose the value in 
the middle of this range as the point estimate of prob- 
ability. Agreement between experts was measured by 
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Table 1 Ranking of questions on the questionnaire 
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the intraclass correlation coefficient (ICC) with 95% 
confidence interval [12]. 

Analyses were performed with SAS software version 
9.2 (SAS Institute, Cary, NC). ICC was calculated using 
the %INTRACC macro written by R.M. Hamer (http:// 
www.psych.yorku.ca/lab/sas/intracc.htm) . 

Results 

Characteristics of the panel of experts 

A panel of 23 experts participated in the two Delphi 
rounds. Seventeen out of 23 participants were hepatolo- 
gists, 2 were infectiologists and 4 did not report their 
medical specialty. Median age of participants was 
38 years of age (IQR: 34-53), with an average of 15 years 
of medical practice (IQR: 10-24). All experts except two, 
for whom this variable was missing, reported that at least 
10% of their patients were HCV-positive. Most experts 
were from Cairo (18/23 - 78%). Others were from Ismailia 
(2/23 - 9%), Banha (2/23 - 9%) and Tanta (1/23 - 4%). 

Estimates of transition probabilities 

To estimate transition probabilities, for 33% of the ques- 
tions, experts gave a range of values in response to ques- 
tions. For other questions, they responded by choosing a 
single value. 

Table 2 presents values of mean transition probabilities 
estimated by the expert panel at the end of the second 
round with 95% confidence intervals. Figure 1 represents 
mean transition probabilities for each question at the 
first and second round in the same order as in the 



questionnaire. A logical trend was observed for each 
value: risks increased with age. Moreover, the mean 
probability of transition (corresponding to death prob- 
ability or aggravation of liver disease probability) was 
consistently higher for men than for women, although 
the difference was not always significant. In addition, prob- 
ability of death increased with the different stages of the 
disease: probability of death was between 10.1% and 26.4% 
in compensated cirrhosis, between 18.0% and 39.4% in case 
of first decompensation, between 20.0% and 39.2% for a 
stable decompensated state, between 27.6% and 54.5% for 
a progressive decompensated state and between 26.6% 
and 61.5% for patients with HCC. The same type of grad- 
ation was observed for risk of occurrence of HCC. 

However, confidence intervals surrounding transition 
probabilities were found to be wide. For example, widths 
of intervals ranged from 4.7% to 12.9% for death prob- 
ability in compensated cirrhosis (first year), from 10.8% 
to 14.6% for death probability in stable decompensated 
cirrhosis and from 14.7% to 20.7% for death probability 
in a progressive decompensated state, all ages and genders 
confounded. Scatter plots in Figure 2 show responses of 
each participant as a function of each question. It illus- 
trates the general wide variability of participant responses 
to the different questions; variations in responses for ques- 
tion 15, for example, ranged from 5% to 90%. 

Based on the intraclass correlation coefficient, vari- 
ability between experts for the entire questionnaire was 
estimated at 0.33 [95% CI: 0.26-0.43] for the first round 
and 0.51 [95% CI: 0.43-0.61] for the second. Although 
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Table 2 Main results of the second round of Delphi - mean transition probabilities (%) with 95% CI estimated at the 
second round of Delphi, and comparison with results from the literature (italics) [8,9,13,14] 

Gender = male Gender = female 
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We assumed that Child-Pugh score class B corresponds to stable decompensated cirrhosis and that class C corresponds to a progressive decompensated state. 



relative convergence appeared to occur between the 
two rounds, disagreement remained high. The minimum 
expected value for the intraclass correlation coefficient 
with high agreement between participants was evaluated 
at 0.75 [12]. 

We performed an analysis stratifying experts by 
age: <40 years of age (13/23) vs. > 40 (10/23). Figure 3 



presents results of this analysis with mean probabilities 
of transition for men (a) and women (b) estimated at 
the end of the second round. White circles represent 
the mean answer for the <40-year-old group and black 
dots for the > 40-year-old group. A difference seemed to 
emerge between the two subgroups of experts. Younger 
experts were, in general, more pessimistic, with higher 
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Figure 1 Mean estimation of transition probabilities by round for men (a) and women (b). Questions are ranked by current stage, transition 
stage and age. Corresponding questions are found in Table 1 . 



probabilities of progression of liver disease and death. In 
addition, estimates of the intraclass coefficient differed 
between the two groups, although they were not statisti- 
cally significant (ICC for the < 40-year-old group: 0.54 
[CI 95%: 0.47-0.65]; ICC for the > 40-year-old group: 0.46 
[CI 95%: 0.37-0.56]). 

Discussion 

There exist no data on the natural history of HCV dis- 
ease in developing countries, and Egypt is no exception. 
We therefore conducted this Delphi analysis to specific- 
ally estimate progression of patients with cirrhosis to 
HCC and death in that country. 

The natural history of HCV disease has been esti- 
mated for developed countries. We assume that, in 
Egypt, management of HCV-infected patients with cir- 
rhosis differs from that of developed countries. More- 
over, patient characteristics and co-morbidities are not 
necessarily similar in these settings, and it was for these 



reasons that we conducted our study. However, we were 
surprised at differences in estimations in our Delphi 
study compared to those reported in developing countries 
[8,9,13,14]. Transition probabilities for some stages and, in 
particular, early stages of cirrhosis, seemed unexpectedly 
high in our study (i.e. annual death probabilities for pa- 
tients with compensated cirrhosis were 10- to 24-fold 
higher in our study than that found in the medical litera- 
ture in developed countries [8]). In contrast, transition 
probabilities for other stages such as late stages of cirrho- 
sis seemed to be lower in our study (i.e. annual death 
probability for patients with first cirrhosis decompensation 
were 1- to 2-fold lower in our study [8]). 

Moreover, we found very strong variability between 
the responses of the different participants. Certain tran- 
sition probability estimates ranged from 5% to 90%. In 
addition, the intraclass correlation coefficient estimating 
overall degree of agreement between experts was low. 
This lack of consensus calls into question the estimates 
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Figure 2 Scatter plot of individual transition probability estimates after the second round of Delphi for men (a) and women (b). 

Corresponding questions are found in Table 1. 



obtained by the Delphi process, especially as other stud- 
ies were conducted in the past to estimate transition 
probabilities by eliciting expert opinions, with similar 
results. Soares et al. attempted to estimate transition 
probabilities for a cost effectiveness transition model of 
negative pressure wound therapy for severe pressure ul- 
ceration [15]. The authors observed variability in experts' 
answers, and felt that this type of result is desirable, since 
it ensures that all views are represented. However, the de- 
sign of that exercise was not a Delphi study, which specif- 
ically aims to obtain a consensus between experts. Schultz 
et al. applied the Delphi technique to lung cancer progres- 
sion [6], with estimation of 5-year survival probabilities by 
14 experts. They did not make any comparison with ex- 
pected values for this disease because of the absence of 
empirical data, but high variability in responses between 
experts was observed. The authors concluded that the dif- 
ferent beliefs may explain variations between practitioners 
in management of patients with solitary pulmonary 



nodules. They mentioned small sample size as one of the 
study limitations and stated that such a result - that is, 
variability in opinions - may not be generalized. Lubell 
et al. conducted a Delphi survey to estimate transition 
probabilities in the natural history and medical man- 
agement of malaria and acute febrile illness [5]. 
Twenty-one panellists participated in that study. They too 
noted wide dispersion on several questions, ranging from 
5% to 100%, and overall lack of agreement between ex- 
perts. Our survey suggests that this type of variability can 
be found for other diseases. This is worrisome, since erro- 
neous perception of risk of disease progression by practi- 
tioners may influence medical choices concerning these 
diseases as well. In a broader context, Cahan et al. were 
interested in the estimation of probabilities by physicians 
in the context of a "threshold approach" for decision- 
making in medicine [16,17]. They asked physicians to 
assess the probability of various diagnoses from a case 
description in a single anonymous questionnaire. Their 
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results indicated that practitioners generally overestimate 
the probabilities of each diagnosis, resulting in a total of 
more than 100% ("subadditivity") for a non-exhaustive list 
of mutually exclusive diagnoses [17]. Second, they also 
found a wide variability in responses among experts 
[16,17]. They did not use a Delphi process, which specific- 
ally aims to increase agreement between the responders. 
However, the same type of problem is observed. The au- 
thors highlighted the "support theory" of Tversky and 
Koehler [18,19]. This theory claims that the probability 
assigned to a description of an event depends on the de- 
scription: two different descriptions of the same event can 
lead to different estimates. Results of these studies show 
that it is indeed a lack of probabilistic thinking rather than 
the type of probability, or a lack of observation of specific 
cases, that may be involved. 

Other explanations may be proposed for the wide vari- 
ability in responses and errors in transition probability 
estimations. First, the infrequency of events and the rarity 
of profiles may lead to imaginary rather than statistical 



estimates (i.e. cognitive-based estimates) [20]. For ex- 
ample, for patients with compensated cirrhosis, orders 
of magnitude for probabilities are around 1 percent and 
sometimes 1 per mile, and overestimations are clearly 
accentuated for such a state. Under such conditions, we 
presume that events are not frequent enough to be fa- 
miliar to practitioners. Profiles may be too specific to 
allow practitioners a good representation of the situ- 
ation described in the questionnaire. Other potential 
explanations concerning cognitive bias and perception 
of risk are present in the literature [21,22]. Kahneman 
et al. and Tversky et al, in particular, describe several cog- 
nitive biases and thought mechanisms involved in human 
judgment. The "representativeness heuristic" implies that 
probability estimations by humans are not based on statis- 
tical or probabilistic reasoning, but rather on judgment by 
representativeness, i.e. on stereotypes based on the de- 
scription of patient characteristics. This mode of reason- 
ing leads to erroneous evaluations, which are sometimes 
in contradiction with elementary probabilistic properties 
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[23]. Other cognitive biases may occur because of the 
"availability heuristic", which states that reasoning tends 
to be based on immediately available information. Infor- 
mation availability may be influenced by various factors. 
Tversky et al. suggest - among others - salience: "it is a 
common experience that the subjective probability of traf- 
fic accidents rises temporarily when one sees a car over- 
turned by the side of the road" [22]. It is reasonable to 
assume that a similar cognitive bias occurs when we ask 
physicians to estimate event probabilities regarding their 
patients' deaths. Abstraction also has an impact on avail- 
ability, because abstract quantities - such as probabilities 
- are not immediately available and require reasoning. 
Evaluation of such quantities can cause what Tversky 
et al. call "biases of imaginability" [22]: instances are gen- 
erated according to a given rule. In our study, questions 
were ordered by gender, current stage of cirrhosis, transi- 
tion stage and age (Table 1). Linearity of the different 
probability functions of age in Figure 1 suggests that ex- 
perts responded according to a predetermined model (lin- 
ear increase of risk with age) and not according to their 
practical experience. 

In the Delphi survey, the iteration process can theoretic- 
ally be repeated in order to increase the opinion's conver- 
gence between experts. However, this point of view does 
not create a consensus [24], because of the risk of an 
artificial consensus: Delphi increases judgment conver- 
gence, but not accuracy [1,20]. In our study, the relative 
convergence observed between the two rounds does 
not seem to lead responses toward more realistic 
values. In contrast, overestimated values become higher 
(Figure 1). If experts have no idea of the answer at the 
first round, there is no reason why iteration would lead 
to better estimations. In addition, anonymity might en- 
courage experts to answer even if they are uncertain 
[25]. In our study, the mean number of missing values 
decreased between the two rounds, from 2.1 (9%) to 0.4 
(2%). It is reasonable to assume that the difference 
might be explained by some form of group pressure. 

Results mentioned above suggest that recommenda- 
tions be made for solicitation of expert opinions. Delphi 
might be a useful tool for compensating for the lack of 
empirical data in medical research [26]. However, for 
Dalkey, knowledge is more reliable than opinion [7] and 
this is why, in the presence of available data, Delphi is a 
useless technique: at best, results are consistent with em- 
pirical results; if results differ, then empirical estimations 
are a priori better. Moreover, the nature of the informa- 
tion requested must be taken into account. While it is ex- 
pected that information concerning medical practices is 
easily evaluated by a panel of experts, probabilistic mea- 
surements or abstract information can cause cognitive 
bias in estimations [21,22]. Thus, it is recommended that 
Delphi not be used to obtain information on the natural 



history of a disease or survival probabilities. Finally, the 
relevance of the questions is subject to caution. Questions 
that are too precise could paradoxically be counterpro- 
ductive, and the familiarity of experts with the different 
aspects of the questionnaire should be subjected to pre- 
liminary discussion. 

Our study suffers from several limitations. First, data on 
transition probabilities used to evaluate quality of results 
were taken from studies in northern countries. Second, al- 
ternative formulations of questions may have been more 
appropriate for estimating transition probabilities by ex- 
perts without probabilistic background [15,27]. Finally, 
opinion (unlike knowledge) is imperfect and this uncer- 
tainty must be measured in the questionnaire [28] . We left 
two options to the experts when responding to questions: 
either respond by a single value or, if unable to provide a 
single value, respond by giving a range that takes uncer- 
tainty into account. The fact that only 33% of the ques- 
tions were answered in the form of an interval illustrates 
the fact that we may have failed to capture this individual 
uncertainty [15,27]. 

Conclusion 

Elicitation of expert opinions to determine transition prob- 
abilities between different health stages of HCV natural his- 
tory in Egypt seems to be inexact, with substantial disparity 
between experts. Estimation of disease progression prob- 
abilities by practitioners involves cognitive biases that imply 
a lack of reliability of the estimates. Thus, even though 
elicitation of expert opinion is a relatively easy way of com- 
pensating for a lack of empirical data, it does not seem suit- 
able for estimation of probabilities of medical events. 

Additional files 



Additional file 1: Explanation Form. 
Additional file 2: Questionnaire. 



Abbreviations 

CI: Confidence interval; HCC: Hepatocellular carcinoma; HCV: Hepatitis C virus; 
ICC: Intraclass correlation coefficient; IQR: Interquartile range. 

Competing interests 

SDB received grants from Roche, Janssen-Cilag and Schering-Plough and 
consultancy honoraria from Merck and GlaxoSmithKline. GE received funds 
from BioGenesic Phasma and BMS. YY received travel grants, honoraria for 
presentations at workshops and consultancy honoraria from Abbott, Bristol- 
Myers Squibb, Gilead, Merck, Roche, Tibotec and ViiV Healthcare. None of the 
other authors report any association that might pose a conflict of interest. 

Authors' contributions 

YY had the idea for the study. DO, SBD, VC and YY contributed to the 
conception and design of the analysis. GE and MKM took part in the 
selection of experts. GE, MEK, MES, and MKM were part of the panel of 
experts. AM, DO, and AC performed first-round statistical analysis during the 
Delphi process. AC performed the second round and additional statistical 
analysis. All authors contributed to interpretation of data. AC drafted the 
article and all authors critically revised it for important intellectual content. 
All authors approved the final version of the manuscript to be published. 



Cousien et al. BMC Medical Research Methodology 2014, 14:39 
http://www.biomedcentral.com/1471-2288/14/39 



Page 9 of 9 



Acknowledgments 

This study was funded by the French Agence Nationale de Recherche sur le 
Sida et les Hepatites virales (ANRS, http://www.anrs.fr), grant number 12215. 
We would like to express our gratitude to experts in Egypt for giving their 
time and sharing their expertise by participing in the Delphi: Samy Zaky El 
Sayed, Maissa El Raziky, Mohamed A. Mohey, M. Magdi Atta, Basem El Sayed 
Eysa, Mohamed Kamal Shaban, Mohamed Atallah, Mai Esmaiel Mehrez, 
Wafaa Ahmed El Akel, Mohamed Adel, Tamer Mahmoud Elbaz, Ayman Salem 
Amer, Abdelkader Farrage, Mohamed El Ateek, , Mohamed Said Abdel Aziz, 
Hassan Hamdy, Abobakr Mohamed, Emad Abdel Sattar, Ahmed Mohamed El 
Deeb and Mahmoud Ahmed Shedid. 

Author details 

1 1AME, UMR 1137, INSERM, F-75018 Paris, France; Univ Paris Diderot, 
Sorbonne Paris Cite, F-75018 Paris, France. 2 EA2694, Universite Lille Nord de 
France, Lille, France. 3 lnserm U995, Universite Lille 2 - Lille Nord de France, 
Lille, France. 4 Department of Community, Environmental and Occupationa 
Medicine, Faculty of medicine, Ain Shams University, Cairo, Egypt, 
department of Endemic Medicine and Hepatology, Faculty of Medicine, 
Cairo University, Cairo, Egypt. 6 Service des Maladies de I'Appareil Digestif et 
de la Nutrition, Hopital Huriez, CHRU Lille, France. 'National Hepatology and 
Tropical Medicine Research Institute, Cairo, Egypt. Conservatoire National 
des Arts et Metiers, Chaire Sante et Developpement, Paris, France. 9 Unite 
d'Epidemiologie des Maladies Emergentes, Institut Pasteur, Paris, France. 
10 Service de Maladies infectieuses et tropicales, Hopital Bichat Claude 
Bernard, Paris, France. 

Received: 8 March 2013 Accepted: 30 January 2014 
Published: 17 March 2014 



References 

1 . Linstone HA, Turoff M: The Delphi method: techniques and applications 
Reading, Mass: Addison-Wesley; 1975. 

2. Dalkey NC, Helmer-Hirschberg 0: An experimental application of the Delphi 
method to the use of experts. Rand Corp: Santa Monica, Calif; 1962. 

3. Deville W, Greacen T, Bogie M, Dauvrin M, Dias S, Gaddini A, Jensen N K, 
Karamanidou C, Kluge U, Mertaniemi R, I Riera RP, Sarvary A, Soares JJ, 
Stankunas M, Strassmayr C, Welbel M, Priebe S: Health care for immigrants 
in Europe: is there still consensus among country experts about 
principles of good practice? A Delphi study. BMC Public Health 201 1, 

1 1 :699. 

4. Hermanides HS, van Vught LA, Voigt R, Muskiet FD, Durand A, van Osch G, 
Koolman-Wever S, Gerstenbluth I, Smit C, Duits AJ: Developing quality 
indicators for the care of HIV-infected pregnant women in the Dutch 
Caribbean. AIDS Res Ther 2011, 8(1):32. 

5. Lubell Y, Staedke SG, Greenwood BM, Kamya MR, Molyneux M, Newton PN, 
Reyburn H, Snow RW, D'alessandro U, English M, Day N, Kremsner P, 
Dondorp A, Mbacham W, Dorsey G, Owusu-Agyei S, Maitland K, Krishna S, 
Newton C, Pasvol G, Taylor T, Von Seidlein L, White NJ, Binka F, Mills A, 
Whitty CJ: Likely health outcomes for untreated acute febrile illness in 
the tropics in decision and economic models; a Delphi survey. PLoS One 
2011, 6(2):el7439. 

6. Schultz EM, Si Ivestri GA, Gould MK: Variation in experts' beliefs about lung 
cancer growth, progression, and prognosis. J Thorac Oncol 2008, 
3(4):422-426. 

7. Dalkey NC: The Delphi method: an experimental study of group opinion. Rand 
Corp: Santa Monica, Calif; 1969. 

8. DAmico G, Garcia-Tsao G, Pagliaro L: Natural history and prognostic 
indicators of survival in cirrhosis: a systematic review of 1 18 studies. 
J Hepatol 2006,44(1)217-231. 

9. Davis GL, Alter MJ, El-Serag H, Poynard T, Jennings LW, Davis GL, Alter MJ, 
El-Serag H, Poynard T, Jennings LW: Aging of hepatitis C virus (HCV)- 
infected persons in the United States: a multiple cohort model of HCV 
prevalence and disease progression. Gastroenterology 2010, 

138(23:5 13-521. 521 e51 1-516. 

1 0. Sievert W, Altraif I, Razavi HA, Abdo A, Ahmed EA, Alomair A, Amarapurkar D, 
Chen CH, Dou X, El Khayat H, Elshazly M, Esmat G, Guan R, Han KH, 

Koike K Largen A, Mccaughan G, Mogawer S, Monis A, Nawaz A, Piratvisuth T, 
Sanai FM, Sharara Al, Sibbel S, Sood A, Suh D J, Wallace C, Young K, Negro F: A 
systematic review of hepatitis C virus epidemiology in Asia, Australia and 
Egypt. Liver Int 201 1 , 31 (Suppl 2):61-80. 



1 1 . Cornberg M, Razavi HA, Alberti A, Bernasconi E, Buti M, Cooper C, Dalgard O, 
Dillion JF, Flisiak R Forns X, Frankova S, Goldis A, Goulis I, Halota W, Hunyady B, 
Lagging M, Largen A, Makara M, Manolakopoulos S, Marcellin P, Marinho RT, 
Pol S, Poynard T, Puoti M, Sagalova O, Sibbel S, Simon K, Wallace C, Young K, 
Yurdaydin C, Zuckerman E, Negro F, Zeuzem S: A systematic review of 
hepatitis C virus epidemiology in Europe, Canada and Israel. Liver Int 201 1 , 
31 (Suppl 2):30-60. 

12. Shrout PE, Fleiss JL: Intradass correlations: uses in assessing rater 

reliability. Psychol Bull 1979, 86(2):420-428. 

1 3. DAmico G, Pasta L, Madonia S, Tarantino I, Mancuso A, Malizia G, Giannuoli G, 
Pagliaro L: The incidence of esophageal varices in cirrhosis. Gastroenterology 
2001,20(5):A2. 

14. DAmico G, Morabito A, Pagliaro L, Marubini E: Survival and prognostic 
indicators in compensated and decompensated cirrhosis. Dig Dis Sci 

1986, 31(5):468-475. 

15. Soares MO, Bojke L, Dumville J, Iglesias C, Cullum N, Claxton K: Methods to 
elicit experts' beliefs over uncertain quantities: application to a cost 
effectiveness transition model of negative pressure wound therapy for 
severe pressure ulceration. Stat Med 201 1, 30(19):2363-2380. 

16. Cahan A, Gilon D, Manor O, Paltiel O: Clinical experience did not reduce 
the variance in physicians' estimates of pretest probability in a 
cross-sectional survey. J Clin Epidemiol 2005, 58(1 1):121 1-1216. 

17. Cahan A, Gilon D, Manor O, Paltiel O: Probabilistic reasoning and clinical 
decision-making: do doctors overestimate diagnostic probabilities? 
QJM 2003, 96(10)763-769. 

18. Tversky A, Koehler D: Support theory: a nonextentional representation of 
subjective probability. Psychol Rev 1994, 101:547-567. 

19. Rottenstreich Y, Tversky A: Unpacking, repacking, and anchoring: 
advances in support theory. Psychol Rev 1997, 104(2):406-41 5. 

20. Murphy MK, Black NA, Lamping DL, McKee CM, Sanderson CF, Askham J, 
Marteau T: Consensus development methods, and their use in clinical 
guideline development. Health Technol Assess 1998, 2(3):l-88. 

21 . Kahneman D, Slovic P, Tversky A: Judgment under uncertainty: heuristics and 
biases. Cambridge. New York: Cambridge University Press; 1982. 

22. Tversky A, Kahneman D: Judgment under uncertainty: heuristics and 
biases. Science 1 974, 1 85(41 57):1 1 24-1 1 31 . 

23. Tversky A, Kahneman D: Extensional versus intuitive reasoning: the 
conjunction fallacy in probability judgment. Psychol Rev 1983, 
90(4)293-315. 

24. Powell C: The Delphi technique: myths and realities. J Adv Nurs 2003, 
41(4)376-382. 

25. Sackman H: Delphi critique; expert opinion, forecasting, and group process. 
Lexington, Mass: Lexington Books; 1975. 

26. Graham B, Regehr G, Wright JG: Delphi as a method to establish 
consensus for diagnostic criteria. J Clin Epidemiol 2003, 56(12):! 150-1 156. 

27. van Noortwijk J, Deker A, Cooke R, Mazzuchi T: Expert judgment in 
maintenance optimization. IEEE Trans Reliab 1992, 41:427-432. 

28. O'Hagan A: Uncertain judgements: eliciting experts' probabilities. Chichester: 
Wiley; 2006. 



doi:1 0.1 1 86/1 471 -2288-1 4-39 

Cite this article as: Cousien ef al:. Is expert opinion reliable when 
estimating transition probabilities? The case of HCV-related cirrhosis in 
Egypt. BMC Medical Research Methodology 2014 14:39. 



Submit your next manuscript to BioMed Central 
and take full advantage of: 

• Convenient online submission 

• Thorough peer review 

• No space constraints or color figure charges 

• Immediate publication on acceptance 

• Inclusion in PubMed, CAS, Scopus and Google Scholar 

• Research which is freely available for redistribution 



Submit your manuscript at 
www. biomedcentra I .com/su bmit 



o 



BioMed Central 



